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NOTE TO: 
FROM : 
SUBJECT: 


9 November 1983 


■\ -'it: 


Director of Central Intelligenc 
Deputy Director for Intelligence 
Hughes Study on Collection Systems 



Attached is the study we have been discussing. 

I have outlined some key points in the first dozen 
pages and call your attention to further highlighting 
on pages 17, 33, 59, 71, 78, and 92. The summary 
is useless. 

One point brought out in my brie^Tn^jfthat I 
did not find in the report is that in assessing 
alternative collection systems you would select 
a half dozen or so samples like the surface to 
air missile sample discussed in the study to give 
you a feel for comparative advantages of systems 
in diverse areas. For example, you might want to 
look at a particular economic target, two or three 
other kinds of military targets and perhaps a 
politicaUmilitary target to give you a relative 
sense of the advantages and disadvantages of each 
of the candid collection systems. You obviously 
cannot apply this cumbersome methodology to all 
possible targets. Sampling* would be required. 


I should caution again that Hughes acknowledges 
more work remains to be done on this^ aSC^ that it is 
cumbersome and time consumingj and my impression is 
that they are continuing this work. 
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FOREWORD 


This interim report was prepared by the Hughes Aircraft Company, 
Fullerton, California under contract 82-N7880000. The prinicipal ^Hughes 
analysts for this effort have been Ms. K.J. Baldasari, Mr. R.S. McKinnell, 
and Mr. W. Lund. ‘ .rTj- 
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I. PROBLEM STATEMENT 





PROBLEM STATEMENT 


HUGHES 


DEVELOP A METHODOLOGY TO PERFORM UTILITY ASSESSMENTS OF 
COLLECTION CONCEPTS 


- THE METHODOLOGY SHOULD BE CAPABLE OF DEALING WITH 
THE ENTIRE CONCEPT LIFE CYCLE 

• RELATE THE METHODOLOGY TO SPECIFIC INTELLIGENCE QUESTIONS 



The purpose of this effort is to develop a methodology to assess the 
utility and capabilities of collection system concepts. Such a methodology is 
needed to bring a systematic procedure to a process whi'ch has heretofore been 
intuitive and variable. 

The methodology developed must be flexible, enabling it to be applied to 
the wide range of concerns and problems with which collection systems must - 
deal. The final task of this project therefore is to show the developed 
methodology's flexibility and adaptability by applying the outlined procedures 
to a mutually agreed upon problem, assessing the information required to 
determine the impact of a new surface-to-air missile (SAM) on the balance of 
power. 
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steo in methodology delineation involved documenting 

tL past Slth techniques which are audltahle. generate traceable analysis 
Se needli“a°Itand^dlied procedure Is especially acute given the 

^hr:e::h:d:“"hf.^:Soi:i:srsh:ufd^”rhiish"^ 

is inteitain Ld possesses a probability distribution whose fom may depend 
u^orthrerpert pLformlng the analysis or formulating the question 
Furthermore the data employed does not remain constant, but instead is 
dvnlmiraS depends upon the timing of the analysis and its requirement. 
Therefore to reflect this diversity the methodology should be adaptable to 
reapplication and easily Incorporate additional information as it become 

available. 
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II. METHODOLOGY DEVELOPMEOT 



DECISION ANALYSIS 


HUGHES 


DECISION ANALYSIS COUPLES THE QUANTITATIVE 
TECHNIQUES OF SYSTEMS ANALYSIS WITH THE 
QUALITATIVE THEORIES OF DECISION THEORY 

DECISION ANALYSIS IS SUITED TO PROBLEMS 
CHARACTERIZED BY 

— UNCERTAINTY 


— COMPLEX PREFERENCES 


- IMPORTANCE 

- UNIQUENESS 

- LONG RUN IMPLICATIONS 





Decision analysis refers to a body of knowledge and techniques which 
combines decision theory and system analysis. Decision theory investigates 
the manner in which people assemble information to make decisions and the 
actual decision making process. System analysis brings to the effort 
techniques for problem decomposition, handling uncertainty in parameters, and 
mathematical decision rules. 

The combination of these two disciplines is ideally suited to the 
collection systems problem since it allows for uncertainty in parameter 
evaluation, can handle complex problems through decomposition, and provides a 
systematic framework for evaluation. Decision analysis is not appropriate for 
smaller problems as the effort of analysis may be too large for less 
significant issues. Therefore, the techniques are most appropriate when the 
Issues under study are complex, of relative Importance, unique and 
identifiable, and the long-run implications are such that a temporarilly 
stable method of review is desirable. 
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SUCCESSFUL DECISION ANALYSIS 
APPLICATIONS 

\ 

: HUGHES : 

• VOYAGER SPACE PROJECT 


• NUCLEAR POWER PLANT SITE SELECTION 

• URBAN DEVELOPMENT 


• FEDERAL ENERGY POLICY 


• CORPORATE INVESTMENT POLICY 


• DAM SITE SELECTION 




^ 


Decision analysis is a proven, developed approach through which 
decisions have been made. Decision analysis has been applied on projects 
ranging from the highly quantitative Voyager Space Project to the qualitative 
concerns of urban development. The broad range of successful applications 
illustrates the flexibility of the method as well as its maturity. 

Selection decisions have been made successfully using these techniques. 
By its very nature, decision analysis is well suited to comparing alternatives 
and making incremental assessments. 
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DECISION ANALYSIS: THE PRO'S 


HUGHES 


• THE DECISION MAKER IS ENCOURAGED TO SCRUNTINIZE THE PROBLEM 
AS AN ORGANIC WHOLE 


• ALLOWS EACH EXPERTTO GIVE TESTIMONY ABOUT A PARTICULAR AREA 
OF EXPERTISE IN AN UNAMBIGUOUS, QUANTITATIVE MANNER THAT 
CAN BE INTEGRATED INTO THE OVERALL ANALYSIS 

• ALLOWS THE DISTINGUISHING OF THE DECISION MAKER'S 
PREFERENCES FOR CONSEQUENCES FROM JUDGEMENTS ABOUT 
UNCERTAINTIES 


• CAN BE USED TO CLEARLY COMMUNICATE THE RATIONALE FOR THE 
ADOPTED STRATEGY AND RALLY SUPPORT FOR IT 


• CLEARLY POINTS OUT THE FACTORS INCORPORATED INTO THE 
ANALYSIS 

I 

• ALLOWS FOR THE DECOMPOSITION OF THE PROBLEM INTO ITS BASIC 
PARTS SO ISSUES WHERETHERE ARE FUNDAMENTAL DISAGREEMENTS 
CAN BE EXPOSED 

• MAY REVEAL PREVIOUSLY HIDDEN RELATIONSHIPS/STRUCTURE 


)fO«7»-7|».27U| 



J 


Decision analysis brings several advantages to a collection system 
analysis. The techniques encourage separation of the factors and basis of the 
decision from the personal biases and feelings of the decision maker. 
Additionally, structure is introduced to the problem which allows for easier 
communication of ideas and improved analysis possibilities. The framework is 
such that sensitivity analysis is possible. Under the intuitive methods of 
the past such analysis of variance was not possible. 

Decision analysis makes complex problems easier because decoi^osition 
into smaller problem pieces is possible. Decomposition allows those most 
capable of performing a subtask to do so without also requiring performance 
outside any areas of expertise. Synergy will result when the pooled opinions 
of these experts are generated which will lead to a better basis for reaching 
decisions than otherwise would have been available. 
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DECISION ANALYSIS: THE CON’S 




* methodology OOESHDT ENSVRE all 

PROBLEM ELEMENTS ARE CONSIDERED 


withoufu': is „„t 

recognized going into a task <jr, n, f chnique, these limitations need to be 

first u.uftt„? or^ci^^n^ iiiTsli r.T.Tniiru 

whacsvsr sapertlss Is available and Mkroptl“rr,;se'‘of 

Structured from the ground since the problem is 

are. however. easily\"^"eeted Ltef if t^ -e possible. These omissifs 
discovered. This weakness also nn-inf^\^^ 

work on the problem to assure full coverage f f If f f 
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METHODOLOGY CHOSEN 
DECISION ANALYSIS 




HUGHES 


OBJECTIVE 


DECISION ANALYSIS SOLUTION 


• STANDARDIZED 
PROCEDURE 


• TREE STRUCTURE SUITED TO WIDE VARIETY OF 
PROBLEMS WITH MULTIPLE OBJECTIVES 


• GENERATE TRACEABLE 
PATHS 

• INCORPORATE MULTI- 
FACETED DATA AND 
INFORMATION 


• TREE ANALYSIS HIGHLIGHTS CRITICAL PATHS 
AND PARAMETERS OF PROBLEM 


• TREE STRUCTURE HANDLES QUALITATIVE AND 
QUANTITATIVE DATA 

• UNCERTAINTY IS EASILY HANDLED BY 
SENSITIVITY ANALYSIS 

• EASILY UPDATED WITH NEW INFORMATION 






To verify decision analysis is a desirable methodology, a comparison is 
needed between the outlined objectives and the advantages of decision 
analysis. From the above chart, it is apparent decision analysis satisfies 
each of the outlined criteria and is therefore suited to the task outlined. 

The remainder of this report addresses the specifics of applying the 
methodology to collection system problems. To do this a simple problem of 
information collection and evaluation will be used for illustrative purposes. 
By working a simple problem in detail, it is felt a more in-depth delineation 
of the methodology is obtained and problems which might be expected are- 
highlighted and resolved. 

The result of using this methodology is the introduction of structure 
and a standardized procedure to what has previously been an intuitive, 
unstructured analysis. The methodology will not replace or reduce the 
importance of expert opinion. The expert's opinion will, however, be more' 
visible and available for review and discussion. 
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THREE PHASE APPROACH 




HUGHES 


II. 


III. 


DETERMINATION OF TREE STRUCTURE AND THE 
ASSIGNMENT OF BRANCH WEIGHTS 

DEFINITION OF COLLECTION CONCEPT AND THE 
DETERMINATION OF COLLECTION UNCERTAINTIES 

APPLICATION OF OTHER FACTORS AND SENSITIVITY 
ANALYSIS 

• TIMELINESS 

• SURVIVABILITY 




The basic approach of the defined methodology divides into three 
phases. The first phase involves formulating the problem to be addressed in 
an information tree structure. This "tree" depicts the relationships and 
hierarchy of the problem from the top of the tree, the question to be 
answered, to the bottom of the tree, the collectable data items which combine 
to answer the overall issue. The intermediate stages of the tree form the 
linkages between collectable data and the overall question. Information at 
these levels might include system parameters, doctrine considerations, ,ahd 
deplo 3 ment questions. 

Once the structure has been outlined, weights are assigned to the 
branches to indicate a particular branch's importance in answering the 
overriding issue at the next highest level or node. A value between O' and 1 
is used to define this importance, 0 indicating no relevance (the branch could 
be eliminated from the tree) and 1 indicating this branch is the only relevant 
branch (all others emanating from this node could be eliminated). 
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the tree structure complete and the weights assigned, the second 

capabiuSes the collection concepts along with their 

capabilities. Defining the concept involves establishing the probabilities 

Thp described at the lowest levels of the tree. 

The probability assessment will in practice take the form of a distribution 

definition wJth successful collection. By combining the concept 

iafifof infii^aM^ ^ obtained which delineates the 

. available to answer the overriding question of the tree 
Incorporation of outside factors not Included in the tree structure or 

avallablliii''ii^M ^^/‘'''““P^^shed in phase three. Considerations such as the 
allabllity or timeliness of a concept, or the availability of data from a - 

given concept versus suother, can be Included In the aLlysL through t"-use 
Of a discount factor. By allowing for the time value of information i^thir 

Add?M* timing of availability differ can be compared. 

Additionally, concepts whose expected lifetimes differ can be analyzed in a 
similar manner. The additional information available because one system will 
have a longer lifetime than another can be handled by "compounding" tSrvIiue 
obtained by the tree and concept definition analysis. The result a cMevlfSy 

" probability distribution showing thf 
availability and its likelihood of collection. Using these 

aversion minlmum"r''%"^''^ concepts can be evaluated keeping in Lnd risk 
aversion, minimum requirements or other factors. 
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III. ILLUSTRATIVE EXAMPLE OF METHODOLOGY 


EXAMPLE PROBLEM 


HUGHES 


JOHN IS CONSIDERING BUYING A HOUSE. HE HAS 
DRIVEN PAST ONE ON THE WAY TO WORK THAT LOOKS 
LIKE A GOOD POSSIBILITY 

HE NOW HAS TO FIND INFORMATION ON HOW WELL THIS 
HOUSE FITS HIS NEEDS — BOTH OBJECTIVE AND 
SUBJECTIVE 


HE WANTS TO KNOW 

• WHAT IS THE BEST WAY TO COLLECT THE 
INFORMATION HE NEEDS TO MAKE A DECISION 

STEP NO. 1: BUILD AN INFORMATION VALUE TREE 

STEP NO. 2: ASSESS THE VALUE OF THE INFORMATION 
THAT CAN BE GATHERED BY COMPETING INFORMATION 
COLLECTION CONCEPTS 

STEP NO. 3: EVALUATE OTHER FACTORS IMPACTING 
COLLECTION PERFORMANCE 






What follows is an in-depth discussion of the methodology which will 
describe why it can be such a powerful decision support tool and, more 
specifically, 

the techniques necessary to apply it, 

- the information it can provide, 

the insight it will give to both analysts and decision makers. 

As a supplement to this technical discussion, a simple example will be used to 
illustrate the key concepts of the methodology. Please note, however, 

the example is by necessity somewhat over-simplified in order to 
demonstrate the concepts; it is not intended to be completely 
realistic, and 

so that it can provide insight into how such an analysis is done, 
the example will be presented in a very detailed and numerically 
intensive manner. 
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The example centers around John who is considering buying a house. He 
has driven by one on the way to work that looks like a good possibility and 
now must collect Information on how well this particular house fits his 

needs. Specifically he wants to know: ' 

"What is the single most effective method to use to collect the 
information necessary to make this decision." 

The basic steps John will need to follow to perform his analysis are: 

(1) Build an information value tree to delineate the raw data that 
should be collected, as well as the relative value of each piece to 

the main question. w f 

(2) Assess collection concept performance by evaluating the ability ot 
each candidate collection concept to meet each raw data requirement. 

(3) Incorporate any other relevant factor and perform sensitivity 
analyses. 


Note: This example has been specifically designed to illustrate all facets of 

the methodology and the reader should not be concerned that the techniques 
seem laborious and involved. In actual analyses many of the steps Illustrated 
will not be necessary because sensitivities of the problem will allow some 
simplification. 
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III. A. INFORMATION TREE DEVELOPMENT 



INFORMATION VALUE TREE 


HUGHES 


REPRESENTS THE RELATIONSHIP BETWEEN DIFFERENT LEVELS OF 
INFORMATION NEED AND THE DATA ELEMENTS THAT SUPPORT THEM 

DEVELOPED BY DRIVING DOWN ONE LEVEL OF INFORMATION AT A TIME 
FROM THE DECISION MAKER'S QUESTION TO THE AVAILABLE DATA 

DEPICTED GRAPHICALLY BY 

• NODES — REPRESENTING SOME QUANTUM OF INFORMATION 

• BRANCHES - REPRESENTING EITHER 

— THE TIE UP TO THE NEXT HIGHER LEVEL OF INFORMATION 
REQUIRING THIS PIECE OF INFORMATION 
— THE TIES DOWN TO THE LOWER LEVELS OF INFORMATION 
ELEMENTS SUPPORTING THIS PIECE OF INFORMATION 

ESTABLISHES THE RELATIVE VALUE BETWEEN ALL ELEMENTS OF 
INFORMATION RELATIVE TO EITHER 

• A PARTICULAR NODE 

• A PARTICULAR INFORMATION LEVEL 





y 


I 

An information value tree is a construct for relating different levels 

[ of information requirements and the raw data supporting them. It is a 

systematic structuring of how these information and data items relate. The 
information value tree is essentially a graph with 

branches - which represent either data or pieces of information 
[ aggregated from lower levels of Information or raw data, and 

nodes - which represent the tieing together of the data or 

aggregated information to support a specific higher level 
I information need, that is, a further aggregation of 

' Information. 

The purpose of such a structure is to establish the value of each information 
I need or data item relative to the particular information requirement (node) it 

I supports as well as to any other information need or data item within the tree. 

In structuring the Information tree, the nodes employed can either be 
"or" or "and" in character. The node type selected will depend upon the 
I structure and Information required by the developer. Whichever structure type 

provides the greatest visibility to the developer should be employed. The 
only precaution needed for the actual analysis is the numerical combinations 
I performed at each node must be consistent with the node type. For "and" nodes 

j all branches into a node should be- summed yielding the nodal result and for 

or nodes only the branches which are active, actually employed or satisfied, 

I should be summed. 
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At the top of the next page is the tree John developed to evaluate which 
collection concept provides the most effective support relative to his main 
question; should he consider buying this house. This question is called the 
"decision maker question." From the tree it can be seen that for John, the 
utility or value of a house is solely a function of 

- whether it fulfills certain critical needs he has established, 

the extent to which the house and yard will need to be cared for, and 

- the investment required and the return expected. 

These are the "critical issues" that drive the decision maker's question. In 
turn, each critical issue is defined by a set of "problem characteristics." 

For example, the investment potential has been broken down into the immediate 
affordability, the continuing affordability, and the future worth. The 
problem characteristics are themselves derived from "data items" which relate 
to pieces of information that are actually collected. Note that the complete 
set of data items that support the decision maker's question are on the far 
right side and range from the number of bedrooms in the house to the planned 
developments in the local area. 

In general, the key concepts to consider at each level of the tree are 
as follows: 

Decision Maker Question - the main question against which the collection 
concepts are being evaluated. 

Critical Issues - the key issues or subquestions that drive the resolu- 
tion of the decision maker question. These are still from a 
qualitative vs a quantitative perspective. 

System/Problem Characteristics - quantifiable information that defines 
the key drivers behind the critical issues. 

Data Items - data that is directly collected or directly derived from 
collected data. 

These levels are guidelines to help the analyst in systematically 
constructing the tree structure such that all the relevant issues are 
addressed and the key data requirements are delineated. The structuring 
process can be done solely by the decision maker, as in this example, or in 
concert with a group of experts in the various areas spanned by the tree and 
is somewhat of an art. It is critical to be as systematic as possible; all 
the relevant information needs to be captured and, conversely, as little as 
possible of the extraneous. A key sign that some part of the tree's structure 
is essentially complete occurs when review of that section produces 
discussions revolving around semantics which generate no new insight or data . 

The information represented at different levels of the tree ranges from 
very qualitative at the decision maker question level (e.g., should John * 
consider buying this house) to very quantitative (e.g., the number of bedrooms 
in the house). Specifically, quantitative information can be described using 
numbers; examples might be the number available, dollars spent, percent lost, 
distance traveled, probability of occurrence. Qualitative information, on the 
other hand, is sometimes referred to as "intangible"; examples might be margin 
for safety, ability to predict, impact on the status quo, level of 
performance. One of the strengths of this decision analysis approach is the 
ability to tie together such different types of data. 
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JOHN'S TREE 


SIG ENOUGH 


THIS HOUSE? 


FUNCTIONAL 

NEEDS 


FULFILLED 

LOCATED WELL 


RELATIVE TO 


MAINTAINABILITY 

SUPPORT 


RECUIREO 

DURABILITY 


IMMEDIATE 


AFFORDABILITY 

INVESTMENT 

CONTINUING 

POTENTIAL 

AFFORDABILITY 


FUTURE WORK 


NUMBER OF BEDR 
^ TOTAL SQUARE FOOTAG E 
SCHOOLS 
WORK 
SHOPPING 

AMOUNT OF YARD WOR K 
(age OF HOUSE 

CONDITION OF SIMILAR HOUSES 

DISASTER VULNERABILITY 
IFIRE. EARTHQUAKE, ETC | 

DOWN PAYMENT 


BASIC MONTHLY PAYMENT 

riNTEFtEiT VARIAKIi 

PROJECTED 

POSSIBLE ASSESSMENTS 

EXPECTED APPRECIATION 
I PLANNED DEVELOPMENTS 


DECISION MAKER CRITICAL PROBLEM 

QUESTION ISSUES CHARACTERISTICS 


DATA ITEMS 


LEVELS OF THE TREE STRUCTURE 


I. DECISION MAKER QUESTION 

• ULTIMATE QUESTION OR PROBLEM TO BE 
ADDRESSED BY REMAINDER OF THE TREE 

II. CRITICAL ISSUES (KEY SUBQUESTIONS) 

• QUALITATIVE QUESTIONS WHOSE SOLUTIONS ARE 
NECESSARY TO ANSWER DECISION MAKER 
QUESTION 

III. SYSTEM/PROBLEM CHARACTERISTICS 

• QUANTITATIVE SYSTEM PARAMETERS WHICH 
COMBINE TO PROVIDE AN ANSWER TO CRITICAL 
ISSUES 

IV. RAW AND DERIVED DATA 

• QUANTITATIVE DATA WHICH IS COLLECTED IN ITS 
RAW FORM BY COLLECTlOIVl SOURCE OR EASILY 
DERIVED FROM SUCH DATA 



QUALITATIVE 

PERSPECTIVE 


QUANTITATIVE 

PERSPECTIVE 
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As John's 

of branches; this is not a requirement, however. More complex questions where 
more detail is necessary or even just desired may have several tiers of 
branches and nodes at each level. Additionally, the number of tiers or 
branches at a particular level does not have to be consistent throughout the 
tree. For example, instead of just stopping with a branch corresponding to 
the relative distance of the house to work, John could have put a node at the 
end of that branch and added two new branches that corresponded to the 
distance relative to where he works and relative to where his wife works. The 
more detailed the branch structure becomes, the more information it captures 
and the more insight it provides into the data and issues that drive the 
decision maker's question. Care must be taken, however, to avoid excessive 
detail; for example, the specific distance to every single school in the local 
area is not of any concern to John, so to break the tree down further in this 
area is undesirable. 
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• THE SET OF BRANCHES SHOULD CHARACTERIZE ALL 
THE SUPPORTING INFORMATION NECESSARY TO BE 
CONSIDERED AT A GIVEN NODE 


SIGNIFIGANCE 

• EACH BRANCH SHOULD REPRESENT AN IDENTIFIABLE 
AND MEASURABLE ELEMENT OF CONTRIBUTING 
INFORMATION 

FAMILIARITY 

• EACH BRANCH SHOULD BE UNDERSTANDABLE TO THE 
EXTENT IT IS POSSIBLE TO ESTABLISH A WEIGHTING 
RELATIVE TO THE OTHER BRANCHES AT THE SAME 
NODE 

INDEPENDENCE 


>»0*T» X« |»-27 Ml 



TWO BRANCHES FROM THE SAME NODE SHOULD NOT 

- REPRESENT THE IDENTICALLY SAME INFORMATION 
REQUIREMENT 

- BE DEPENDENT ON EACH OTHER 



In developing an Information value tree, there are some simple 
analytical guidelines for delineating the set of branches at a particular 
node. Since each node represents the tieing together of all its relevant 
data, the set of branches must be "complete”, that is there must be a branch 
for each contributing piece of Information. For example, "immediate 
affordability" is completely defined by what the down payment and the closing 
costs are. By the same reasoning, a branch should only be constructed if it 
represents a "significant" element of information, that is, it has some- 
relevance to the information requirement represented by the node. The fact 
that the house has a tile roof is not relevant to whether it is big enough. 
Next, each branch should be "familiar", that is, it should represent a piece 
of information that both the analyst and reviewer can understand and 
evaluate. Last, all the branches should be "independent"; no two branches 
should depend directly on one another or represent the identically same piece 
of information. 
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III. B. DELINEATION OF INFORMATION TREE WEIGHTS 


DELINEATION OF WEIGHTS 


HUGHES 


JOHN HAS NEVER BOUGHT A HOUSE BEFORE SO 
HE IS NOT SURE WHAT THE RELATIVE 
IMPORTANCE OF THE VARIOUS ISSUES ARE 

12. INSIGHT. HE CONSULTS SOME OF HIS FRIENDS 
(WHO^HE^B|UEVES TO BE KNOWLEDGEABLE IN THE AREA 

MARY: WHO IS IN THE PROCESS OF BUYING HER THIRD 
HOME 

SUE: WHO HAS EXTENSIVE REAL ESTATE 

INVESTMENTS 

GARY: WHO HAS NEVER BOUGHT A HOUSE. BUT WHO 
JUST FINISHED A SEMINAR FOR FIRST-TIME 
HOME BUYERS 


tH7 I3| 


Once the structure of the information value tree has been established 
(the nodes and the branches have been defined), the next step is to delineate 
the relative value weightings. This can be done by the decision maker, in 
this case, John, or it can be delegated by the decision maker to a group of 
experts. Since John has virtually no experience in the real estate market, he 
decided he needed help in assigning the weights to the branches of his tree. 

So he selected three experts to advise him. 

Mary is in the process of buying her third home so she's h^d a fair 
amount of experience not only in actually buying a house, but also 
in maintaining one. Of all his "experts", she probably represents 
the one whose experience John feels is the most consistent with his 
perspective. 

Sue is very much Involved in the real estate market and has many 
investments. She is predominately Interested in investment 
potential. 

Gary has never bought a house but he just finished attending a 
seminar designed for first-time home buyers so he has information 
about what the "experts" say should be considered when buying a 
house. 
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The weight assigned to each branch reflects its value or contribution to 
the information requirement relative to the other branches emanating from the 
same node. This weight may range in value between 0 and 1 where 

0 - the piece of information has no impact on the information 

• requirement at all, and 

1 - this piece of information is the only contributing information 

required. 

The weights assigned to all the branches at a particular node must sum to 1, 
reflecting the premise that together they represent the complete set of 
significant information needed to define the information requirement at that 
node. 

The actual calculation of the branch weights is a two step process. 
First, the relative value of all the branches at a particular node is 
established. For example, piece B of information may be twice as Important as 
piece A and piece C of Information may be three times as important as piece 
A. This can be analytically described by: 

3*B = 6*A = 2*C 

The next step is to normalize by forcing the weights to sum to 1 which gives: 

A+B+C=A+ 2*A + 3*A = 1 
or 

A = 1/6 

B = 1/3 

C = 1/2 

By requiring the branch weights to sum to 1 at every node, a consistent 
value structure is forced on the entire tree, enabling the analyst to readily 
calculate the relative value of any piece of information or data item within 
the tre.e. Ultimately, this leads to a preference ranking over the set of 
collectables (data items) which can then be used to evaluate the set of 
potential collection concepts. 
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ASSIGNMENT OF RELATIVE VALUES 



PROCESS WHEREBY THE RELATIVE VALUE OF THE 
VARIOUS PIECES OF INFORMATION (BRANCHES) 
REQUIRED TO SUPPORT A PARTICULAR (HIGHER) 

LEVEL OF INFORMATION (NODE) ARE ESTABLISHED 

VALUES (OR WEIGHTS) MAY RANGE BETWEEN 0 AND 1 
WHERE 

0 — HAS NO IMPACT AT ALL 

1 — IS THE ONLY CONTRIBUTING FACTOR 

AT A PARTICULAR NODE (QUANTUM OF INFORMATION) 
THE WEIGHTS ASSIGNED TO ALL THE CONNECTING 
BRANCHES (SUPPORTING INFORMATION) MUST SUM 
IDENTICALLY TO 1 TO ALLOW COMPARISON OF 
INFORMATION REQUIREMENTS AT NODES OF THE SAME 
AND HIGHER LEVELS 




CALCULATION OF BRANCH WEIGHTS 


HUGHES 


GOALS: 

• THE WEIGHTS REFLECT THE RELATIVE VALUE OF THE SUPPORTING 
INFORMATION TO DEFINING THIS PIECE OF INFORMATION 

• ALL THE WEIGHTS CORRESPONDING TO THE SUPPORTING 
INFORMATION MUST SUM IDENTICALLY TO 1 

PROCESS: 

• ESTABLISH THE RELATIVE VALUE OF THE VARIOUS INFORMATION 
REQUIREMENTS, EG 

VALUE (B) = 2* VALUE (A) 

VALUE (C) = 3*VALUE (A) 

OR 

3B = 6A = 2C 

• NORMALIZE TO FORCE WEIGHT TO SUM TO 1, EG (AS, ABOVE) 

A + B + C = 1 
A + 2A + 3A = 1 
6A = 1 

A = 1/6. B = 1/3, C = 1/4 


Approved For Release 
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r 

\ 

SAMPLE OF EXPERT OPINIONS 

: HUGHES : 



mahv sue gaby 




Three very different sets of weights were provided by John's experts as 
shown in the trees above. Mary and Sue, for example, have divergent 
priorities for the "investment potential" in general and the "future worth" in 
particular. Mary is much more concerned about the supportability of the house 
and the functionality it provides; Sue, with her background in real estate 
investments, concentrates on the investment section of the tree. Gary has yet 
another set of weights although his are closer to Sue's. 

The next issue for John is how to aggregate these three different 
perspectives into a single consensue tree. 
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The derivation of consensus weights for the decision maker's tree 
requires some procedure for aggregating the values for the weights provided by 
experts into a single set of estimates. Just summing the values up and taking 
the average, although a simple approach, does not take into account any 
information the decision maker may have about the differing levels of 
expertise the experts may have or any knowledge of their individual biases. 
Additionally, such a procedure provides no way to adjust the impact extreme 
opinions may have on the aggregation (either to emphasize or to mask them). 

The question of how best to create a consensus position from a pool of 
expert opinions has received much attention from decision analysis experts. 
Many procedures have been proposed varying from the simplistic, such as the 
averaging approach suggested above, to the theoretically sophisticated; as 
yet, however, there is no general agreement about which approach yields the 
most valid results. However, to be useful for the collection systems problem, 
an aggregation procedure should meet four requirements: 

(1) There should be a way to incorporate any preferences the decision 
maker may have between the experts so differing levels of emphasis 
can be assigned in proportion to the perception of their expertise. 

(2) In order to preserve the relative value structure of the entire 
tree, the aggregation procedure must produce consensus weights such 
that the sum of the values assigned to the set of branches at any 
node is still exactly 1. 

(3) The weight structure for any particular expert must be Incorporated 
in such a way that his or her relative rankings are preserved. 

(4) To enable the procedure to be as usable as possible, the 
information required to implement it should be readily available. 

An example of one of the more analytic approaches that has been 
developed is a procedure proposed by Morris. It is a very rigorous Bayesian 
approach that uses each expert's past performance to condition, that is 
weight, his or her current prediction. Initially, the excellent theoretical 
foundation for this procedure, as well as others of a similar nature, seemed 
to strongly point to the implementation of such an approach within this 
methodology. l^pon further investigation, however, the very strength of such 
procedures proved to also be their weakness, at least relative to the type of 
decision support methodology that is needed to handle the collection system 
problem. The application of approaches such as Morris's requires access to a 
reasonably large sized data base describing the performance of the expert with 
similar problems. Typically, such a data base is not available. As a result, 
the approach selected for implementation with this methodology is less 
sophisticated but in return the decision maker will have more control of how 
the consensus is to be achieved. If over time the data base necessary to 
support a method such as Morris' can be accumulated, a more sophisticated 
approach can be added to the procedure suggested in this report. 
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AGGREGATION OF EXPERT OPINION 


HUGHES 


COMBINING THE OPINIONS OF SEVERAL EXPERTS IN A 
VALID MANNER ONE NEEDS TO CONSIDER 

• DIFFERING LEVELS OF EXPERTISE BETWEEN 
EXPERTS 

• INDIVIDUAL BIASES 

• EFFECTS OF EXTREME OPINIONS 

METHODOLOGY CHOSEN MUST ALLOW FOR THE 
ADJUSTMENTS NECESSARY AND ALSO BE 
SATISFACTORY IN THE 

• TEMPORAL 

• REASONABLE IN THE AMOUNT OF DATA 
REQUIRED 

• EASILY TRANSFERABLE AND AT THE SAME TIME 
MAINTAIN AS MUCH DATA AS POSSIBLE 


MORRIS APPROACH 


HUGHES 


• APPROACH TO EXPERT RESOLUTION OR 
CONSENSUS DISCUSSED IN THE PROPOSAL 

• RIGOROUS BAYESIAN APPROACH 

• BASED ON PRIOR PERFOMANCE OF EXPERTS 

• DRAWBACK 

— DATABASE ON PRIOR PERFORMANCE NOT 
TYPICALLY AVAILABLE 

• APPROACH SELECTED 

— LESS TECHNICAL 

— GIVES THE DECISION MAKER MORE DIRECT 
CONTROL OF THE CONSENSUS PROCESS 
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For the collection systems problem, the approach selected to derive the 
consensus estimate from a set of expert opinions is based on a weighted 
average of the experts' opinions. Given the estimates of N experts for a 
value to be assigned to a particular weight, 

V(l), V(2), ..., V(N), 

as well as a set of preferences (or weights) from the decision maker 
associated with each of these experts, 

W(l), W(2), ..., W(N), 

the consensus value (W*) is the normalized weighted average of the experts 
opinions, that is, the weighted average divided by the sum of the weights, 

* 1 N 

W = *[ 2 W * V ]. 

N „ 3=1 j i 

i=l i 

This approach, besides being very straightforward to apply, is appealing in 
the way it allows the decision maker to emphasize the estimates of the various 
experts in a way that is consistent with his or her evaluation of each of the 
experts. Another benefit is that it does not require the extensive historical 
database needed to implement an approach like Morris's, In fact, if the 
decision maker has no preference between the experts, the weights, W^, can 
be set identically to 1. 

John has decided that of his three experts, Mary's perspective is the 
one that most closely matches his position. As such, he feels that her 
estimates should be weighted about twice as much as either Sue's or Gary's, 
whose opinions he feels should be weighted about equally. The resulting 
preferences or weights are 
Maryi 2 

Sue : 1 

Gary: 1 

Using these values, John calculates the consensus weights for his information 
value tree. As an example, the expert opinions associated with the branch for 
“functional needs" are 
Mary: 0.35 

Sue: 0,15 

Gary: 0.25 

giving a consensus weight of 

W* = (l/4)*[(2)(0.35) + (1)(0.15) + (1)(0.25)] 

= 0.275 

The complete set of consensus estimates for the weights are shown in tl\e 
adjoining table. 
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CONSENSUS METHODOLOGY 



HUGHES 





GIVEN 

• A SET OF WEIGHTS OR RELATIVE VALUES PROVIDED 
BY N EXPERTS. 

V-j , Vj^ 

• A SET OF DECISION MAKER PREFERENCES OR 
WEIGHTS ASSOCIATED WITH THOSE N EXPERTS. 

W,^ 

THEN 

• THE CONSENSUS WEIGHT. W*. IS DEFINED BY 

W* = ^ ^ 

N 2 W.V. 

2 Wj 1 1 i 

i=1 

EXAMPLE 

• IF V-i = 0.35. V2 = 0.1 5. V3 = 0.25 

W-i = 2. W2 = 1 . W3 = 1 

• THEN W* =± (2) (0.35) + (1 )(0.1 5) + (1 ) (0.25) = 0.275 

4 



EXPERT PREFERENCESAVEIGHTS 


FUNCTIONAL NEEDS 
BIG ENOUGH 
BEDROOMS 
SO FOOTAGE 
LOCATION 
SCHOOLS 
WORK 
SHOPPING 


SUPPORT REQUIRED 
MAINTAINABILITY 
AMT OF YARD WORK 
AGE OF HOUSE 
DURABILITY 

CONDITION OF SIMILAR 
DISASTER VULNERABILITY 


INVESTMENT POTENTIAL 
IMMEDIATE AFFORDABILITY 
DOWN PAYMENT 
CLOSING COSTS 
CONTINUING AFFORDABILITY 
BASIC MONTHLY 
INTEREST 

PROJ. ASSESSMENTS 
FUTURE WORTH 
EXP. APPRECIATION 
PLANNED DEVELOPMENT 


RELATIVE 

MARY SUE GARY CONSENSUS TO 
TREE 


0.15 0.25 

0.30 0.40 
0.40 O.SO 
0.60 0.50 

0.70 0.60 
0.60 0.45 

0.10 0.20 
0.30 0.35 


0.25 
0.S0 
0.40 
0.60 
0.30 O.SO 
0.70 0.60 

0.30 0.40 


O.SO 
0.40 
O.SO 
0.50 
0.40 
0.35 
0.40 0.40 

0.20 0.25 

0.60 0.20 
0.60 0.50 

0.40 O.SO 




CALCULATION 

OF 

CONSENSUS 

WEIGHTS 
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The consensus tree developed by John is shown on the adjoining page at 
the left in the top figure. As required, the weights associated with the 
branches at each node sum to 1. John has reviewed this tree and is fairly 
comfortable with both the structure and the weights. 

On the right of the figure is another version of the same consensus tree 
which shows the relative value of each piece of information to all other items 
at the same level instead of just those items at the same node. The 
"maintainability” issue, for example, is evaluated to be three times as 
important as the question of the house's "future worth . For this version of 
the tree, the values are calculated by multiplying the series of branch 
weights from the original tree along the tree from the decision maker question 
to that specific branch. In the case of the "maintainability" branch, the 
weight is calculated by multiplying the weight for that branch times the 
weight for the "support" branch, that is, 

W = (0.700)*(0.376) 

= 0.2622 

This approach to structuring the information value tree provides a 
ranking at each level of all the corresponding information requirements. This 
is particularly useful at the data item level as it produces a prioritization 
of the collection requirements. In John’s tree, the most important data item 
to be collected, the branch with the high relative value is the "age of the 
house" (0.164); at the other end of the spectrum, the least important data 
item is "possible assessments" (0.028). This modified form of the tree makes 
it readily apparent that the chief issue seems to be the support of the house 
and, more specifically, the maintenance of the house, whereas the least 
concern is given to its future worth. 

This completes the first step of the analysis, the development of an 
information value tree. The decision maker now has insight into the data 
items that would be desirable to collect as well as their relative 
importance. This ranking will be useful in the next step, the evaluation of 
the various candidate collection concepts. 
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III. C. COLLECTION CONCEPT DEFINITION AND DATA DEVELOPMENT 


INFORMATION COLLECTION 
CONCEPTS 


HUGHES 


NEXT STEP: ASSESS THE PERFORMANCE {I.E., THE VALUE 
OF THE INFORMATION COLLECTED) OF THE VARIOUS 
INFORMATION COLLECTION CONCEPTS TO BE STUDIED 

ACCOMPLISHED BY ASCERTAINING EACH CONCEPT'S 
ABILITY TO COLLECT EACH REQUIRED DATA ITEM IN 
TERMS OF 

• EXPECTED LIKELIHOOD OF COLLECTING THE 
REQUIRED DATA 

• THE MINIMUM LIKELIHOOD OF COLLECTION 

• THE MAXIMUM LIKELIHOOD OF COLLECTION 

WHERE THE LIKELIHOOD OF COLLECTION IS A FUNCTION 
OF 

• THE PROBABILITY OF COLLECTING THE DATA 

• THE QUALITY OF THE DATA COLLECTED. I.E.. THE 
ABILITYTO SUPPORTTHE SUCCEEDING INFORMATION 
REQUIREMENTS 


>M47»3a 


The next step of the methodology focuses on the evaluation of the 
ability of each candidate concept to meet each raw data collection 
requirements.^ This, combined with the relative weight structure derived in 
the previous step, will provide a means for analyzing collection performance. 

For each concept, the collection ability relative to each data 
requirement will need to be specified not only in terms of its expected or 
likely performance, but in order to be able to do confidence assessments^ 
minimum and maximum performance levels will also need to be delineated. “All 
performance likelihoods are a combination of both the probability of 
collection and the quality of what is collected relative to the data 
requirement. For example, relative to information about "planned 
developments" in the local area, if one of John's candidate concepts on the 
average will have an 80% chance of collecting some data, and if collected, the 
data will probably cover 95% of the developments, then a likely performance 
level, P, would be 

P = .80 * .95 

or approximately 75%. Quality thresholds, i.e., stipulation that if the data 
is not at least a certain quality it is not any good at all, are handled by 
assigning a 0 to the quality factor if it falls below the threshold. 
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John has identified four candidate approaches to collect the data he 

needs: 

(1) Drop by the house in question and talk with the owner. 

(2) Call up the city for information. 

(3) Arrange with a friend (who is a real estate agent) to get the 
necessary data. 

(4) Check the newspaper to see if the owner is advertising the house. 
There may be other approaches that he might use, but these four are the ones 
under consideration. The question then is which one is the most effective 
collection concept, which one will collect the information with the highest 
overall potential value. 

The first step in evaluating any of these collection concepts is to 
define the concept's ability to collect each data item in terms of its best or 
maximum performance, its worst or minimum performance, and its likely or 
expected performance. Data on the performance of the concepts can be provided 
by the decision maker or by an expert or experts designated by the decision 
maker. In the case of more than one expert, the same consensus techniques 
used to delineate the branch weights can be applied. 

The table on the adjoining page summarizes the performance data for 
John's problem for each of his collection concepts, For example, in the area 
of "disaster vulnerability", John's estimate is that the owner's likelihood of 
supplying him that information is somewhere between 40% and 60%, the city's 
likelihood is between 60% and 80%, and his friend's likelihood is between 50% 
and 70%. He does not expect to find this particular data item in a newspaper 
advertisement. 
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POTENTIAL SOURCES FOR 
JOHN'S DATA 


HUGHES 


JOHN HAS IDENTIFIED 4 POTENTIAL WAYS TO COLLECT 
THE DATA HE NEEDS: 

(1 ) DROP BY THE HOUSE IN QUESTION AND TALK WITH 
THE OWNER 

(2) CALL UP THE CITY FOR INFORMATION 

(3) ARRANGE WITH A FRIEND WHO IS A REAL ESTATE 
AGENT TO GET THE NECESSARY DATA 

(4) CHECK THE NEWSPAPER TO SEE IF THE OWNER IS 
ADVERTISING THIS HOUSE 

HE NOW NEEDS TO DEFINE THE PERFORMANCE OF EACH 
OF THESE RELATIVE TO EACH DATA REQUIREMENT IN 
TERMS OF MINIMUM. EXPECTED, AND MAXIMUM 
PERFORMANCE 

• CAN BE DONE BY THE DECISION MAKER OR BY USING 
EXPERT CONSULTATIONS AND CONSENSUS 
TECHNIQUES AS WAS DONE WITH THE WEIGHTS 


INFORMATION PERFORMANCE 
FACTORS 



HUGHES 
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Once the information has been gathered on the performance range of each 
candidate collection concept, the next step is to evaluate their expected 
performance. For each information requirement, this is expressed in terms of 
an expected information value which is determined by multiplying the expected 
performance level by the corresponding relative weight from the modified 
information value tree. In the case of the third collection concept (talking 
with his friend) and the first collection requirement (the number of bedrooms), 
relative value of information = 0.075 
expected performance =0.9 
expected information value = (0. 9)*(0.075) 

= 0.0676 

The expected information values are in turn summed over the entire tree to 
provide the overall expected information value for the complete tree for each 
concept. These can be compared to determine the collection concept with the 
highest expected performance. 

The adjoining table contains the expected performance figures for each 
of the collection concepts in John's problem. The first and the third 
concepts (talking with the owner and talking with his friend) have markedly 
higher expected performance than the remaining candidates. The margin between 
the top two, however, is close enough that it is difficult at this point to 
differentiate between them; not enough is known, only the expected performance 
has been compared, not the complete performance range. To do this, John will 
need to move into the third phase of the evaluation provess, the sensitivity 
analysis. 
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'N 

PERFORMANCE ANALYSIS 

: HUGHES : 


THE INITIAL CONCEPT ANALYSIS IS BASED ON THE EXPECTED 
PERFORMANCE OF EACH CANDIDATE COLLECTION CONCEPT 
AND USES 


— EXPECTED PERFORMANCE LIKELIHOODS 

— CONSENSUS WEIGHTS 


BASICALLY INVOLVES WEIGHTING THE EXPECTED 
PERFORMANCE LIKELIHOODS WITH THE WEIGHTS (RELATIVE 
TO THE DATA ITEM LEVEL) ASSOCIATED WITH THE 
INFORMATION REQUIREMENTS 


^ 

r 

EXPECTED PERFORMANCE 


HUGHES 



INFORMATION SOURCE 

EXP. VALUE 

TALK WITH THE OWNER 

0.58 

TALK WITH THE CITY 

0.23 

TALK WITH FRIEND (AGENT) 

0.60 

CHECK NEWSPAPER 

0.15 


^ ^ 
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III. E. SENSITIVITY ANALYSIS 



SENSITIVITY ANALYSIS 


HUGHES 


• SENSITIVITY ANALYSIS IS THE CRUCIAL STEP OF THE 
METHODOLOGY 


• PROVIDES DECISION MAKER WITH THE 
QUANTITATIVE LIMITS OVER WHICH HIS/HER 
QUALITATIVE DECISION WILL BE VALID 

• ANALYSIS CAN BE APPLIED SEVERAL PLACES 

• OVERALL TREE 

• INTRALEVEL ON THE TREE 

• BETWEEN CONCEPTS 

• INTERNAL TO CONCEPT ANALYSIS 

• METHODS 





• WORST CASE/BEST CASE 

• MONTE CARLO 



The sensitivity analysis is perhaps the most critical phase in the 
methodology because it provides the decision maker insight into the limits 
over which the collection concept selected is the most effective. Finding the 
concept with the highest expected value for the information collected is only 
the first part of the solution; it is equally important to understand the 
range over which it is the most effective solution and how sensitive the value 
of the information collected is to variances in each concept's performance 
relative to the various information requirements. 

Classically, sensitivity analysis refers to procedures in which 
parameters are varied over their possible values in order to determine the 
degree of variation in the resulting solution. Here sensitivity analysis will 
be used in two ways 

- to evaluate the impact on the overall information value over the 
entire range (not, just the expected value) of each collection 
concept’s performance relative to the various information 
requirements, and 

to incorporate factors in addition to the value of information that 
differentiate the performance of various collection concepts. 

Note: The reader is once again warned that the example illustrated here has 
been specifically designed to illustrate all aspects of the developed 
methodology. In actual practice, not all the steps shown will be 
necessary as each problem will possess possibilities for simplification. 
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JOHN'S PERFORMANCE 

^ 

ANALYSIS 

: HUGHES : 




TALK WITH 
OWNER 

TALK WITH 
CITY 

TALK WITH 
FRIEND 
(AGENT) 

r'“— ' 

CHECK 

NEWSPAPER 


MIN 

MAX 

MIN 

MAX 

MIN 

MAX 

MIN 

'max 


NUMBER OF BEDROOMS 

0.58 

0.58 

az3 

0.23 

0.60 

0.61 

0.15 

0.15 



SQUARE FOOTAGE 

0.58 

0.58 

a23 

0.23 

0.59 

0.62 

0.13 

0.17 



SCHOOLS (LOCATION) 

0.57 

0.59 

0.23 

0.23 

0.59 

0.61 

0.14 

0.16 



WORK (LOCATION) 

0.58 

0.58 

0.23 

0.23 

0.59 

0.61 

0.15 

0.15 



SHOPPING (LOCATION) 

0.57 

0.59 

0.23 

0.23 

0.60 

0.61 

0.15 

0.15 



AMOUNT OF YARD WORK 

0.56 

0.59 

0.23 

0.23 

0.60 

0.60 

0.15 

0.15 


ij 

AGE OF HOME 

0.57 

0.59 

0.20 

0.26 

0.57 

0.64 

0.12 

ai8 



CONDITION OF SIMILAR HOMES 

0.56 

0.59 

0.23 

0.23 

0.59 

0.62 

0.15 

0.15 



DISASTER VULNERABILITY 

0.58 

0.58 

0.23 

0.23 

0.60 

0.61 

0.15 

0.15 ■ 



DOWN PAYMENT 

0.58 

0.58 

0.23 

0.23 

0.60 

0.61 

0.15 

0.15 



CLOSING COSTS 

0.58 

0.58 

0.23 

0.23 

0.59 

0.62 

0.15 

0.15 



BASIC MONTHLY PAYMENT 

0.58 

0.58 

0.23 

0.23 

0.60 

0.61 

0.15 

ai5 



PROJECTED INTEREST VARIANCE 

0.58 

0.58 

0.23 

0.23 

0.60 

0.61 

0.15 

0.15 



POSSIBLE ASSESSMENTS 

0.58 

0.58 

0.23 

0.23 

0.60 

0.61 

0.15 

0.15 



EXPECTED APPRECIATION 

0.58 

0.58 

0.23 

0.23 

0.60 

0.61 

0.15 

0.15 



PLANNED DEVELOPMENTS 

0.57 

0.59 

0.23 

0.23 

0.60 

0.61 

0.15 

0.15 



V ^ ^ 


The first step of a more in-depth analysis of the performance of the 
various candidate collection concepts involves investigating the changes in 
the overall value of the information collected at the limits of the 
performance ranges associated with the various information requirements. The 
purpose of such an analysis is to determine 

the impact each collection requirement has over its performance 

range on the value of information collected, and 

the degree to which the various concepts overlap, that is, are- 

indistinguishable. 

The effect of the performance of each concept relative to each information 
requirement is studied by re— calculating the overall value of the information 
collected by 

— fixing the performance of the requirement under consideration first 
at its minimum and then its maximum performance level, and 

— using the expected value performance estimates for all other 
information requirements. 

This provides the analyst insight into the factors that determine which 
concepts are the most effective. 
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The results of such a performance analysis are summarized In the 
adjoining table. Each entry represents the Impact relative to a particular 
collection concept of evaluating a specific Information requirement at either 
the minimum or maximum (versus expected) level of concept performance. There 
are a couple of observations of note. 

~ The Impact of some of the information requirements seems to be 
Independent of whether collection concept performance is fixed at 
its minimum, expected, or maximum level of performance. Examples 
are : 

~ number of bedrooms 

- location relative to work 

- disaster vulnerability 

- down payment 

- basic monthly payment 

- projected Interest variance 

- possible assessments 

- expected appreciation 

When this occurs, a more detailed analysis to determine the exact 
way in which collection concept performance varies between its 
minimum and its maximum levels is unnecessary; it's adequate to fix 
its performance at the expected level. 

The four candidate concepts have different ranges of information 


value : 

(1) Talking with owner 56-59% 

(2) Talking with the city 20-26% 

(3) Talking with friend 57-63% 

(4) Checking newspaper 12-18% 


The range of values for concept If2 and for concept //4 in no way 
overlaps that of any of the other concepts. However, concept //I and 
//3 overlap considerably. Usually talking with the friend seems to 
have a higher return than talking with the owner, but not always 
(see "age of house" for example). 
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INCORPORATION 


OF OTHER FACTORS 


HUGHES : 



THE EVALUATION OF COLLECTION CONCEPT PERFORMANCE 

MUST INCLUDE TWO OTHER FACTORS 

• TIMELINESS 

• SURVIVABILITY 

TIMELINESS 

• THE IMPACT ON THE VALUE OF THE INFORMATION 
COLLECTED DUE TO THE DELAYS IN IMPLEMENTING 
THE CONCEPT OR DELAYS IN THE CONCEPT OF 
PRODUCING THE DATA 

SURVIVABILITY 

• THE IMPACT ON THE VALUE OF THE INFORMATION 
COLLECTED DUE TO THE LIKELIHOOD OF THE 
COLLECTION CONCEPT ENDURING OR SURVIVING 

BOTH ARE FUNCTIONS OF EITHER 

• THE DECISION MAKER'S PERCEPTIONS, OR 

• THE CONSENSUS RESULT OF GATHERING THE 
OPINIONS OF EXPERTS SELECTED BY THE 
DECISION MAKER 

J 


The evaluation of collection concept performance must include two other 
factors: timeliness and survivability. While the overall value of the 

information collected is the key discriminant between concepts, that value can 
be impacted by delays in implementing it or by the length of time over which 
it continues to provide information. The extent of that value impact is a 
function of the performance of the concept as well as the decision maker's 
evaluation of its importance. 

A key concern in collection system analyses is the incorporation of. 
revisit times or capabilities into the analyses. Revisit capabilities are 
accounted for in one of two ways in the methodology. The first approach, 
which is the most straightforward and easily applied, involves incorporating 
revisits into the collection concept performance assessment. With this 
approach the concept is viewed over time with the assessment of capability 
including several collection time intervals or opportunities. In simple 
terms, the question to be answered would be: Given the concept will visit 

this often and have these attempts at collection, what is the probability you 
will satisfy this particular information need? In this sense the concept 
capability includes performance over time and the details of individual visits 
or opportunities are suppressed in favor of an overall performance assessment. 
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The second alternative which can be applied to revisit questions 
involves using discounting methods similar to those discussed under concept 
survivability. In this approach each opportunity is analyzed individually and 
multiple opportunities over time are combined using the discounting methods 
discussed later in this section (see page 46). The application of this method 
involves selecting an appropriate discount factor. 

The selection of either the overall performance assessment or the 
discounted assessment will depend upon analyst confidence and knowledge. The 
ultimate goal, however, is an assessment of overall performance which the 
analyst feels is believable and accurately reflects systems capabilities. 
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TIMELINESS 


HUGHES 


• GIVEN 

V=VALUE OF INFORMATION NOW 

N=AMOUNT OF TIME BEFORE IT CAN BE COLLECTED 

Ft(N)=DISCOUNT FACTOR TO BE APPLIED TO 

INFORMATION VALUE GIVEN A DELAY OF N 
Pt(N)=PROBABILITY OF A DELAY OF N 

• then 

Et (V)=EXPECTED VALUE OF INFORMATION GIVEN 
TIMELINESS CONSIDERATIONS 





= VZFt U) * Pt (j) 

j 

J 


When there is a gap between the point in time the need for some 
information is assessed and when it is actually received, the value of what is 
collected may be reduced. To evaluate collection concepts with different 
delays, the decision maker must specify to what extent delays of various 
durations impact the overall value of the information collected. This impact 
is expressed as a discounting factor on the overall value of information. 

Given 

V = overall value of information relative to some decision 

maker question 

N = number of intervals that the collection of that 

information is delayed 

Ft(N) = timeliness discount factor associated with a delay of N 
intervals 


then 

V^(N) = discounted value of information given a delay of N 

, intervals 


V*Ft(N) 

Typically, the discount factors, F^CN), are expressed as elements in a 
discount stream, i.e., if each period of delay corresponds to "p percent 
loss, then 
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For example, a 5% discount factor for 4 periods would be 

- [tp^] ‘ 

= 0.8227 

Since the decision maker may not be able to predict a priori the exact 
number of intervals that the receipt of information may delayed, there is 
usually a need to specify the probability of delays of various duration and to 
evaluate the expected discounted value of information. 

Given . 

Pt(N) = probability of a delay of N intervals 


Pt(N) 

then 

Et(N) 


expected value of information 'discounted for timeliness 
M 

1 [Vt(j) * Pt(j)] 
j=l 


M 

= V* 2 [Ft(j) * Pt(j)] 

Note that the probability distribution does not need to be specified in a 
detailed way. It is sufficient to grossly estimate it over a handfull of 
points; for example, 

Pt(0) = 0.2 
Pt(l) = 0.4 
Pt(2) = 0.3 
Pt(3) = 0.1 
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SURVIVABILITY 



• GIVEN 

V=VALUE OF INFORMATION NOW 
N=AMOUNT OF TIME OVER WHICH IT WILL BE 
COLLECTED 

Fs(N)=ENHANCEMENT FACTOR TO BE APPLIED 
FOR INFORMATION IN THE Nth PERIOD 
Ps(N)=PROBABILITY OF AN ADDITIONAL N PERIODS 


• THEN 

Es(V)=EXPECTED VALUE OF INFORMATION GIVEN 
SURVIVABILITY CONSIDERATIONS 

k 


= V* I P (k) * [S F (j)] 
k s j=0 * 



When information is collected over an extended period of time the 
of what is collected may be enhanced. To evaluate collection concepts i 
different survivabilities, the decision maker must specify to what 
e‘“nd“fcoUectloa enhances the overall value of the laf ovation collected. 
This impact is expressed as a compounding of value over the numbe 

intervals. 

Given 
V 


decision 


= overall value of information relative to some 

maker question ..Uoi- 

N = number of intervals over which the collection of that 

information continues _ 

Fg(N) = relative value of some information in the Nth interval 


then 

Vg(N) = enhance value of information over N intervals 
N 


= I V*Fs(J) 

j=0 

In other words, the decision maker must 
each additional interval of collection. 


specify the incremental benefit of 
If there is none, then Fg(j) is set 
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to zero. As with timeliness, however, these compounding factors are typically 
expressed as elements in a discount stream, l.e., if each interval of extended 
collection corresponds to a ”p" percent discount factor, then 

F3<n) = ^ 

Since the decision maker may not be able to predict a priori the exact 
number of intervals over which the receipt of information may continue, the 
probability of survival over various durations usually must be specified and 
the expected enhanced value of information evaluated. 

Given 

^s( N) = probability of surviving N intervals 

then 

^s( N) = expected value of information enhanced for survivability 

- f 

- S Vg(k) * Pg(k) 
k=0 '■ 


N , k 

= V * Z P (k) * Z [F (j)] 
k=0 ^ j=0 

Again, the probability distribution does not need to be specified in a 
detailed way. It is sufficient to grossly estimate it over a handful of 
points. For example, 

if additional information is discounted at a rate of 10%, and 
Pg(0) = 0.4 
Ps(l) =0.3 

Ps(2) = 0.2 

Pg(3) = 0.1 
then over 3 years 


E3O) 


3 r k 

V * Z P,(k) * 

1 r\ ^ ® 


= V * |(.4*1) + .3 * (l+j^) + .2 * o) 

,1 * 1 + ^ , + -L- 1 

( 1 . 1 )^ ( 1 . 1 )^ 1 


( 1 . 1 )' 


= 1.8685 * V 
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ADJUSTMENTS TO JOHN'S ANALYSIS 


HUGHES 






IN ORDER TO GET THE INFORMATION FROM HIS REAL 
ESTATE FRIEND. JOHN WILL HAVE TO WAIT AT LEAST 
TWO WEEKS UNTIL HE COMES BACK FROM VACATION 

JOHN FEELS THAT THIS INEFFICIENT USE OFTIME WILL 
REDUCE THE VALUE OFTHE INFORMATION HE COLLECTED 
BY THE FACTORS 

1.050 ~ 

ALL OTHER CONCEPTS HAVE COMPARABLE 
TIMELINESS 
SURVIVABILITY 

THEREFORE THE PERFORMANCE ANALYSIS NEEDS TO 
BE RECONSIDERED 




Tn order to get the information from his friend in real estate, John 
will have to wait at least two weeks until he comes back from vacation. 
this period of time, John feels that the value of the information 
collected will degrade somewhat due to the fact that the house may go 
market. Therefore he estimates that a delay of two weeks wxll discpt the 
value of what is collected by about 5%, for a discount factor, F, ot 

F = 1/1.05 = 0.95 j 1 . 

This third collection concept is the only one expected to have sue a p ay, 

John expects to be able to implement any of the f the 

To adjust John's analysis to reflect this delay in 
third candidate collection concept, what is required is the deduct 
the relative values associated with the information va^ue 

by the time the information is collected it will have been diocounted in 


m 
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by about 5%. For example, the expected information value the third collection 
concept is expected to collect for the first collection requirement, the 
number of bedrooms, is recalculated by 

relative value of information = (1/1,05) * (0.075) ~ 0.072 
expected performance = 0.9 

expected information value = (0.9) * (0.072) =: 0.06A8 
When all of the expected information values have been recalculated, they are 
again summed to determine the discounted expected value for the information 
collected. 


- 49 - 


Approved For Release 2003/09/29 : CIA-RDP86B00269R001 300050001-1 


Approved For Release 2003/09/29 : CIA-RDP86B00269R001 300050001-1 


The adjoining tables show the effect that the discounting had on both 
the performance factors associated with the third concept and on the 
performance analysis as a whole. The expected performance of the third 
concept, talking with the friend who is a real estate agent, typically is 
reduced between 3 and 4 percent across all of the information requirements. 

As for the comparison of the complete set of collection concepts, a check of 
the performance ranges shows that while concepts 2 and 4 are still much lower 
than 1 and 3, the choice between 1 and 3 is even less clear than before. 
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ADJUSTED PERFORMANCE FACTORS 


HUGHES 



NUMBER OF BEDROOMS 
SQUARE FOOTAGE 
SCHOOLS (LOCATION) 

WORK (LOCATION) 

SHOPPING (LOCATION) 


AMOUNT OF YARD WORK 
AGE OF HOME 

CONDITION OF SIMILAR HOMES 
DISASTER VULNERABILITY 


DOWN PAYMENT 
CLOSING COSTS 
BASIC MONTHLY PAYMENT 
PROJECTED INTEREST VARIANCE 
POSSIBLE ASSESSMENTS 
EXPECTED APPRECIATION 
I PLANNED DEVELOPMENTS 


ORIGINAL 


min. I max. 


DISCOUNTED 


min. I MAX. 



PERFORMANCE RANGES 


INFORMATION SOURCE 

TALK WITH OWNER 

TALK WITH CITY 

TALK WITH FRIEND (AGENT) 

CHECK NEWSPAPER 


MIN. 

MAX 

VALUE 

VALUE 

.56 

.59 

.20 

.26 

.54 

.61 

.12 

.18 
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STOCHASTIC DOMINANCE 

\ 

: HUGHES : 

• A CONCEPT IS COMPLETELY DOMINATED IF THERE 
IS AT LEAST ONE OTHER CONCEPT HAVING ALL ITS 
MINIMUM INFORMATION VALUES GREATER THAN ANY 
MAXIMUM INFORMATION VALUE FOR THE 
DOMINATED CONCEPT 

• SUCH CONCEPTS NEED NO FURTHER 
CONSIDERATION 


JVM79 «7.6'2; sai 

J 











In analyzing the competing collection concepts, there occasionally will 
be a case where the performance of one of the candidate concepts is so low 
relative to other concepts that it no longer needs to be considered. A 
concept, A, is said to be stochastically or probabilistically dominated if no 
matter what values the collection performance variables take, there is at 
least one other concept, B, such that for every information requirement, R j , 
the minimum performance range associated with B exceeds the maximum 
performance associated with A. Since the goal is to find the single most 
effective concept, no further consideration needs to be given to a dominated 

concept.* , 

As shown in the preceeding table on performance ranges, concepts I and 4 

are completely dominated by concepts 1 and 3. Therefore John will no longer 

consider concepts 2 and 4 in his analysis. 


*Except if the ultimate use is in cost-effectiveness trade off. However, this 
is not the problem presently being considered. 
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The ultimate goal of this methodology should not be the achievement of a 
stochastically dominant solution. While this is desirable from the decision 
maker s viewpoint, his task is made easier, the non-linear aspects of the 
problem does not quarantee the existence of a dominant solution. Therefore, 
stochastic dominance is not used as a goal for solution selection, but instead 
serves as a means of problem simplification by allowing for the identification 
and elimination of totally unacceptable solutions. If a single, dominant 
solution is not found, the decision maker must then view the decision in terms 
of required levels of performance, as well as risk aversion or acceptance 
preferences. Using these two measures, alternatives which do not dominate 
each other can be distinguished and a solution selected. 
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PROBABILISTIC ANALYSIS 


; HUGHES i 


• A MONTE CARLO APPOACH IS USED TO DETERMINE 
THE COMPLETE PERFORMANCE SPECTRUM FOR EACH 
CONCEPT 


• PROCEDURE 

— PERFORMANCE DISTRIBUTION RELATIVE TO EACH 
DATA REQUIREMENT ARE DEVELOPED FOR EACH 
NON-DOMINATED CONCEPT 

— GIVEN THE RELATIVE VALUE OF EACH DATA 
REQUIREMENT. THE PROBABILITY OF THE 
INFORMATION VALUE OBTAINED FOR EACH 
CONCEPT IS DETERMINED BY A MONTE CARLO 
ANALYSIS USING THOSE PERFORMANCE 
DISTRIBUTION 

— ALLOWS CONFIDENCE ASSESSMENT 


V ^ 


To further compare the remaining concepts, it is necessary to get more 
insight into how the value of the information they collect varies over the 
performance ranges relative to each information collection requirement. This 
is done by developing distributions that describe the probability of achieving 
various values of information. These will allow the analyst to do confidence 
assessments. 

The first step in performing a probabilistic analysis is to roughly 
determine the performance distribution relative to each information 
requirement for each candidate collection concept still under consideration. 
The specification of these distributions can be a simple and as gross as 
merely specifying a handful of points (for example, the minimum performance 
level, the 10th percentile, 30th percentile, the 50th percentile (expected 
performance), 70th percentile, 90th percentile, and the maximum performance 
level). In John's example, all the performance distributions were assumed to 
be uniform, that is, given 

a = minimum performance level 

b = maximum performance level 
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then if P(v) is the probability of collecting information with at least a 
value of V, 

P(v) = (v-a)/(b-a), a s V ^ b 

P(v) = 0, otherwise 

and if p(v) is the probability that the information collected had exactly a 
value of V, 

p(v) = l/(b-a), a ^ V s b 

p(v) = 0, otherwise 

Another distribution which also could have been selected is the normal 
distribution. The key, however, is that it is not necessary to specify a 
"classic" distribution if there is not one that represents the collection 
concept's performance relative to a particular information requirement; a 
simple estimate based on expert judgement is sufficient. 

It is important to note that while these performance distributions can 
be developed for every information requirement, those requirements which the 
performance analysis has shown to have virtually no impact on the overall 
value can simply be left at their expected value. This saves the analyst the 
extra work of defining distributions for those requirements. 

Once all the necessary performance distributions have been defined, they 
are used with Monte Carlo techniques to derive the probabilistic performance 
of the value of the information collected for each remaining candidate 
collection concept. The basic approach is to select samples from the 
performance distributions, weight them with the relative value of the 
associated Information requirement, adjusted to reflect any timeliness or 
survivability considerations, and add them all together to get a sample of the 
distribution for the value of the information collected. More formally, 
given 

Xj(i), the ith sample from the performance of jth information 
requirement , 

Vj, the relative value of the jth Information requirement, 
n, the number of information requirements 

Y^, the ith sample from the information value distribution 

then 

n 

Yi = S Vj * Xj(i) 
j=l 
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The adjoining graphs illustrate the results of the probabilistic 
analysis for John's problem. The top figure is a graph of the probability of 
achieving a guaranteed value of information. Note that although the curves 
are close, there is a difference which influences which one should be 
selected. If John is risk-adverse, he should talk with the owner because 
there is less chance of getting the lowest possible values; if however John is 
risk-immune, he should instead talk with his friend because there is a chance 
he will collect a higher value. 

The differences between the two concepts are even more clearly 
highlighted in the lower figure. Talking with the owner brings more 
consistent results, there is less variance. Talking with a friend could just 
as easily give you a lower information value as a higher relative to that 
which can typically be achieved by the other concept. 
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LIKELIHOOD OF COLLECTION 


HUGHES 



PROBABILITY OF 
AT LEAST 
THIS VALUE 


• TALK WITH OWNER 
TALK WITH FRIEND 



50 0.55 0.60 

RELATIVE VALUE OF INFORMATION 


PERFORMANCE COMPARISON 


PROBABILITY 


TALK WITH OWNER 
TALK WITH FRIEND 



HUGHES 



RELATIVE VALUE OF INFORMATION 
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DECISION MAKER REVIEW 


HUGHES 


AT ANY POINT IN THE ANALYSIS, THE ENTIRE PROCESS 
ACCOMPLISHED TO THAT POINT IS COMPLETELY 
ACCESSIBLE TO 


• REVIEW 

• QUESTIONING 

• RE-ASSESSMENT 


ALL ASSUMPTIONS ARE OUT IN THE OPEN AND THE 
ANALYSIS PATHS ARE READILY TRACEABLE 






One of the strengths of the methodology is the generation of traceable 
paths through the analysis. Since every assumption is documented and the" 
analytic process is straightforward and completely visible, the study is 
accessible at any point for review. This means the decision maker does not 
need to rely solely on an Intuitive understanding of the problem which is 
especially important when the results are to be communicated and perhaps 
defended to others. The methodology itself provides a structure for 
discussing the analysis and the results and, if necessary, for defining at 
What point re-assessment may be necessary. Additionally, if re-evaluation is 
required, any new assumptions can be readily substituted for old with 
retention of much of the remaining structure. 
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To purchase a home, John is going to need financial assistance from his 
parents; so when he finished his collection concept analysis, 
his parents for review. Overall, they understood and agreed with what John 
had done. They did, however, feel that if this house was to be in a sense an 
Investment for them as well, the weights on the information value tree should 
reflect more of an emphasis on the investment issues. Their recommendatio 
was that the consensus tree should be recalculated, but this time with the 

following weights 

3 for Sue's values 
2 for Mary's values 

1 for Gary's values j. ^ ^ 

The revised calculation of the consensus weights is shown in the adjoining 

table. The basic formula used was • 

W*(i) = ith consensus weight 

= (1/6) * [2 * (Mary's ith weight) + 

3 * (Sue's ith weight) + 

1 * (Garv's ith weight) ] 













P 

P 



P 

P 
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REVIEW OF JOHN'S ANALYSIS 


AFTER JOHN COMPLETED HIS ANALYSIS HE SHOWED IT 
TO HIS PARENTS SINCE HE NEEDS THEIR HELP TO 
FINANCE SUCH A PURCHASE 

IN GENERAL THEY AGREED WITH WHAT HE HAD DONE. 
THEY FELT HOWEVER THAT THE WEIGHTS ON JOHN'S 
TREE SHOULD BE ADJUSTED TO REFLECT MORE OFTHEIR 
EMPHASIS ON THE FUTURE WORTH OF THIS INVESTMENT 

THEIR SUGGESTION WAS THATTHEIR JOINTCONSENSUS 
TREE SHOULD BE CALCULATED VVITH 

3 FOR SUE'S WEIGHTS 
2 FOR MARY'S WEIGHTS 
1 FOR GARY'S WEIGHT 


SM47»J7 (4^2? «2( 



RELATIVE 

MARY SUE GARY CONSENSUS TO 

TREE 


EXPERT PREFERENCES/WEIGHTS 


functional needs 

BIG ENOUGH 
BEDROOMS 
SQ FOOTAGE 
LOCATION 
SCHOOLS 
WORK 
SHOPPING 


SUPPORT REQUIRED 
MAINTAINABILITY 
AMT OF YARD WORK 
AGE OF HOUSE 
DURABILITY 

CONDITION OF SIMILAR 
DISASTER VULNERABILITY 


INVESTMENT POTENTIAL 
IMMEDIATE AFFORDABILITY 
DOWN PAYMENT 
CLOSING COSTS 
CONTINUING AFFORDABILITY 
BASIC MONTHLY 
INTEREST 

PROJ. ASSESSMENTS 
FUTURE WORTH 
EXP. APPRECIATION 
PLANNED DEVELOPMENT 



REVISED 

CALCULATION 

OF 

CONSENSUS 

WEIGHTS 
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COMPARISON OF DATA PRIORITIES 


HUGHES 


AGE OF HOUSE 
AMOUNT OF YARD WORK 
SQUARE FOOTAGE 
CONDITION OF SIMILAR HOUSES 

AGE OF HOUSE 
PLANNED DEVELOPMENTS 
AMOUNT OF YARD WORK 
EXPECTED APPRECIATION 

NUMBER OF BEDROOMS 
DOWN PAYMENT 
CLOSING COSTS 

LOCATION RELATIVE TO SCHOOLS 

DOWN PAYMENT 
CLOSING COSTS 

CONDITION OF SIMILAR HOUSES 
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The revised consensus trees, both the basic (with branch weights 
relative to the preceeding node) and the modified (with branch weights 
relative to the level), are presented on the next page in comparison to the 
original consensus trees. Note how the emphasis has moved towards the 
investment issues in general and the future worth in particular. 

In the above table are ranked lists of collection requirements, one 
reflecting the original analysis, one reflecting this revised analysis. Some 
of the collection requirements have been significantly re-ordered, especially 
square footage 

- number of bedrooms 
planned developments 

- expected appreciation 
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Having completed the revised consensus tree, John can now procede to the 
assessment of the potential collection concepts. Since his parents had no 
disagreement with the information performance factors that were developed in 
the original analysis, the revised performance analysis follows much as 
before: relative to each collection concept, the revised relative values of 

information for each information requirement are multiplied by the expected 
performance of the concept and the resulting expected information values are 
summed to determine the overall expected Information value collected. The 
results of this analysis are shown in the top figure of the next page. Once 
again it appears talking with the city or checking the newspaper fall short; 
this time, however, the margin between talking with his real estate friend and 
talking with the owner is greater. So the preliminary assessment is the third 

collection concept appears to be the most effective. 

To investigate this preference more thoroughly, John again considered 
the impact on the overall value of the information collected relative to each 
collection concept by evaluating each information requirement at its minimum 
and maximum levels of performance. The results, shown in the figure at the 
bottom of the next page, indicate that talking with the real estate friend 
appears to stochastically dominate all the other candidate collection 
concepts, however the issue of timeliness remains to be accounted for. 
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As in the original analysis, John must adjust the assessment of the 
third collection concept, talking with the real estate friend, to reflect the 
two week delay in collecting any information. The same estimate is used for 
the discount rate, 5%, which is a factor of (1/1.05) or approximately 0.95. 
This translates into a reduction of about 3 to 4% in the performance factors 
for this collection concept as shown in the figure at the top of the next pap 
A review of the ranges for the performance factors associated with each 
of the collection concepts, adjusted for timeliness consideratpns (pe bptom 
figure on next page), indicates that while the concepts of talkip wip the 
city and checking in the newspaper are indeed stochastically domipted, pep 
is a slight overlap for the other two. So before John completes his analysis, 
he will do a probabilistic analysis on these two remaining concepts. 
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The revised results of the probabilistic analysis are shown in the 
figures on the adjoining page. This time there is no question of which 
concept has the most effective performance. As the top figure shows, talkl g 
with the real estate friend consistently yields a likelihood for collecting a 
hicher overall value of Information. It is Important to note, as the lower 
figure shows, this is completely independent of the range of performance of 
the two concepts, which was not changed; the third concept still has a greater 
possible variance in its performance. Because the relative weights of the 
information collected have changed, however, the overall expected perfomance 
of the two concepts has diverged, leaving the third concept the clear choice. 
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As is readily apparent, the application of this methodology to 
collection systems problems is not a trivial exercise. Typical of all 
decision analysis approaches, it requires the compilation of a significant 
amount of data, including 

- key issues to be Investigated, including the information required to 
support the process and each piece's relative weight 

evaluation of experts 

- collection concept performance — not just at the likely or expected 
level but also minimum and maximum levels and, in some cases, an 
approximation of the entire performance distribution 

- quantification of impacts of discounting effects due to timeliness 
considerations and compounding effects due to survivability 
considerations 

As such, this approach should be reserved for large, resource-critical 
decisions. 

However, the usefulness of much of this data does not end with the 
study. Related problems may be able to use portions of the performance 
estimates or large subsections of the tree either intact or as a template or 
prototype. Similarly, there may be follow-on applications that reconsider 
some of the basic elements and leave the rest essentially the same. Examples 
might be the evaluation of an additional concept or the availability of a new 
expert or the analysis of the same problem but from a different point in time. 
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Instead of initiating an information collection effort at the time the 
original study was done, John just did some preliminary investigation and 
delayed collecting everything else. A month later he decided to go ahead and 
seriously analyze the house in question. But the evaluation he would have 
done before was not completely correct any longer as he had already collected 
a certain amount of Information (such as the number of bedrooms, the square 
footage, the relative distances, etc.). Therefore, collecting it again had 
significantly reduced importance. So he went back, to his experts for updated 
weights. New consensus weights were calculated using the same evaluation of 
the experts as was done for the revised weights 

- 3 for Sue's value 

- 2 for Mary's value 

- 1 for Gary's value 

The updated weights supplied by the experts as well as the new consensus 

weights are shown in the top figure on the next page. These weights produce a 
new ordering of information. Again, a signficant re~ranking has taken place, 

especially for 

- age of house 

- amount of yard work 

- square footage 

- number of bedrooms 

- planned developments 

- expected appreciation 

- disaster vulnerability 

- possible assessments 
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With the updated consensus tree, John procedes to the collection concept 
evaluation phase. Assuming the same set of collection concepts with the same 
performance characteristics, an analysis is done of the updated performance, 
generating overall expected values for the Information collected for each 
concept. The results are presented at the top of the next page. This time 
talking with the friend appears to be significantly more effective than any 
other concept. 

To verify that talking with the friend is the concept to be chosen, John 
considered for a third time the impact on overall value of information 
collected relative to each collection concept from evaluating each information 
requirement at its minimum and maximum levels of peformance. The results, 
shown in the figure at the bottom of the next page, demonstrate unequivically 
that the third concept stochastically dominates the entire set. Therefore, 
there is no need for further analysis. 


- 74 - 


Approved For Release-2^03/99/29^ : ^^- RDP8 6 D00260R001 300050001 = 1 









Approved For Release 2003/09/29 : CIA-RDP86B00269R001 300050001-1 


UPDATED EXPECTED PERFORMANCE 
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IV. DEMONSTRATION EFFORT 



PROBLEM SELECTION CRITERIA 

• CONSENSUS PROBLEM 

* problem mutually agreed 

UPON 

• MUST DEMONSTRATE ADAPTABILITY OF 
METHODOLOGY 

• POSSESS A DEGREE OF COMPLEXITY TO EXERri<5F 

ALL FACETS OF METHODOLOGY EXERCISE 

• ANALYs'is^'^ application OF SENSITIVITY 


V. 


3404 |4.tr-Ul 




For the methodology developed to be of use to the analyst, it must be 
easily applied to the problems with which he/she is concerned. The preceding 
toy example is useful for Illustration and education, but a demonsLation 
effort on a more relevant problem to the collection concept analyst is 
needed. Care should be taken in choosing the topic of this effort so that 
maximum benefit may be achieved. 

of all, the problem should be of enough importance to warrant the 

hfv^ to the community. The problem should 

have sufficient complexity to fully utilize and illustrate the documented 
methodology. Finally, the problem should be such that sensitivity analysis 
can be eeanlagfally employed and be need to shoe the eontlualoS Lfre^'lL 
such an effort makes possible. 

In summary, the demonstration effort should show the power and 
usefulness of the methodology as well as its flexibility and adaptability 
e remainder of this report documents what has been completed to date 
relative to this effort. 
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IV. A. SAM PROBLEM 



The final stage of this analysis involves applying the developed 
methodology to a problem of interest to the intelligence community. By 
documenting this process, an example is available to the analyst to use as a 
guideline when performing additional analyses relating to collection concept 
concerns. 

The problem chosen for illustration involves the information required to 
answer the questions What effect will the introduction of a surface to air 
missile (SAM) have on the balance of power? The point to keep in mind here is 
the frame of reference for the analysis. The methodology does not attempt to 
answer the question regarding the SAM and its effects, but instead the 
methodology determines the information required to answer the question. Given 
these requirements, the ability of a particular collection concept to collect 
the desired information can be evaluated and the rest of the sensitivity 
analysis completed. 

As of *'the "writing of this interim report, the SAM problem has been 
completed through the structuring of the information tree the structure is 
Illustrated on the following pages. The remainder of the analysis will be 
documented in the final report of this study. 
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IV. B. INFORMATION TREE STRUCTURE 
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1 


The top level tree for the SAM problem is shown above. The level of 
information is seen to get more specific with descent through the tree. For 
example, the first division is between direct threats posed by the system and 
modifications that could be undertaken to regain the balance of power. The 
tree expands in detail until the raw or collectable data level is reached. 

The tree structure in its entirety is illustrated on the following pages and 
is indexed using Roman numerals. The numerals refer to the figure number on 
which the continuation of the tree can be found. For example, the expansion 
of the homeland deployment tree can be found on Figure VIII. 
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VIII. HOMELAND DEPLOYMENT 
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XI. CRUISE MISSILE 
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XIII. BOMBER 
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XV. HARD COUNTERS — ARM 
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XVIII. RADAR CAPABILITY (SOLE 
PERFORMANCE) 
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XIX. CAPABILITY (SOLE 
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XX. MISSILE CAPABILITY 
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XXI. SEARCH/ENGAGEMENT RADAR 
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WAVEFORM 

PROCESSING 

STABILITY 


BANDWIDTH 
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PRACTICES 

FREQUENCY 

POWER 
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BEAMWIDTH 


NOISE FIGURE 
SCAN TIME 
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V. SUMMARY 



THE METHODOLOGY DEVELOPED AND DOCUMENTED HAS 
THE FOLLOWING STRENGTHS 


• ADAPTABLE 

• TRACEABLE 

• USES MULTIPLE SOURCES AND TYPES OF DATA 

• REFLECTS DECISION MAKERS POINT OF VIEW FOR AUDIT 
AND REVIEW 

• ALLOWS FOR EXPERT OPINION AGGREGATION 

• SENSITIVITY ANALYSIS 



^ t- fh^s interim report is the methodology which has 

method handles both ^^ed to form a consensus opinion. Most 

several experts opinions to be ^gS^^^ate is „hich provides for 

^t:jz iitz .. co„.ia.nca uvau 

in the conclusions reached. 
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